Papers with State space models
LAMB: A Training-Free Method to Enhance the Long-Context Understanding of SSMs via Attention-Guided Token Filtering (2025.acl-short)
Copied to clipboard
Zhifan Ye, Zheng Wang, Kejing Xia, Jihoon Hong, Leshu Li, Lexington Whalen, Cheng Wan, Yonggan Fu, Yingyan Celine Lin, Souvik Kundu
| Challenge: | Recent work attributes performance degradation to an exponential decay in hidden-state memory. |
| Approach: | They propose a token filtering strategy that is training-free and attention-guided . they propose 'LAMB' to preserve critical tokens during inference . |
| Outcome: | The proposed token filtering improves long-context performance by 30.35% over state-of-the-art methods on benchmarks. |
LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models (2025.acl-long)
Copied to clipboard
| Challenge: | State space models lack interpretability tools for long-context sequence modeling. |
| Approach: | They propose a token-level decomposition method for Mamba-1 and Mamba-2 that enables fine-grained interpretability. |
| Outcome: | The proposed method is able to reveal Mamba’s token-to-token interaction patterns across multiple tasks including translation, copying, and retrieval-based generation. |